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A pulse modulation system is modeled with M wavejorms \s m (t)\" , each 
of which is amplitude scaled and simultaneously transmitted over a single 
physical channel. An infinite pulse train is assumed with signal interval 
T, which is determined by bandwidth consideration of the channel. We 
restrict the receiver to be linear with M outputs, one for each signal wave- 
form. 

At a high signal-to-noise ratio the main sources of interference at the 
input to the receiver are the intersymbol interference and crosstalk; by 
crosstalk we mean the interference between the different waveforms. It 
is desirable, therefore, for the receiver to eliminate both types of interference 
and to minimize the remaining error due to additive noise in the channel. 
This constraint on the intersymbol interference and crosstalk is defined as 
the generalized Nyquist criterion. 

The receiver which accomplishes the above is determined for a mean 
square error criterion. Finally, some examples are presented which de- 
monstrate the ease with which the generalized Nyquist criterion can be 
used to design waveforms without intersymbol interference or crosstalk. 

I. THE MATHEMATICAL MODEL 

The mathematical model for a pulse modulation system is shown in 
Fig. 1. The M waveforms {s m (f)}; u , which are assumed linearly in- 
dependent and of equal energy, are simultaneously transmitted over 
a single physical channel. Information is carried on each waveform by 
amplitude scaling the waveforms s m (t) by the real numbers \a m \" which 
are random variables. An infinite pulse train is assumed with signal 
interval T so that the resulting transmitted waveform is 

EZ« sjl -nT). (1) 
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]Tjg. i _ Model of the pulse modulation transmission system. 

Characterizing the linear time invariant channel by its impulse re- 
sponse, h(t), we define r m (t) as the convolution of s n (t) with h(t) so 
that the received signal waveform is 



Z Y,a nm rM-nT). 



(2) 



To this the channel adds stationary zero mean noise, ri(t), with cor- 
relation function n(r) and spectral density N(f). The received waveform 
is processed by a bank of receivers [w k \" whose M outputs are sampled 
at times t = nT, n = 0, ± 1, ±2, • • • to give b nm which are the estimates 
of the a nm . 

If we consider the set {s m (t)\" with our one physical channel as 
comprising M different channels then we can refer to the interference 
of the waveform due to s k (t) with that of s m (t) (m ^ k) as crosstalk. 

Restricting our attention to linear time-invariant receivers then we 
can characterize the receivers [w k ]" by impulse response {w k (t)\" so 
that the output of the receivers can be expressed as 



oo M r><*> 

h(t) = E £ a, m v mk (t -pT)+ v(x)w k (t - x) dx, 

,, cc m 1 •'-00 



where 



v mk (0 = r Ml ~ •'•)'•«.(•'•) dx. 

J-00 

The sampled outputs are designated by b nk , 

b nk = b k (nT). 



GO 
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At high signal-to-noise ratio (S/N) where 

I>L(0) 

S/N = „ '"" , (6) 

X) / / n(y - x)w k (x)w k (y) dxdy 

t = l •'-00 •'-co 

the main sources of interference at the input to the receivers are inter- 
symbol interference and crosstalk. It is desirable, therefore, for the 
receiver to process its input so that the output eliminates intersymbol 
interference and crosstalk; i.e., that 

Z E <i„ m v mk (nT) = a nk (7) 

j> = — oo m-1 

for all possible sequences of the a nm . This is equivalent to requiring 
that 

v„ lk (til) = 8 mk d„ (8) 

n = 0, ±1, ±2, ••• , 

where the 8 ( j are Kronecker delta functions. Further justification for 
inqiosing this constraint at high S/N is provided in the Appendix. 

We use as our error criterion the mean square error averaged over 
the receiver outputs 

J« = Jj]L i ni(a nm - b yj, (9) 

where the expectation is with respect to the random variables a nk and 
the noise. 

We are now in a position to specify the problem concisely: to de- 
termine the linear receiver which minimizes the mean square error 
under the constraint that there be no intersymbol interference or 
crosstalk. 

II. A GENERALIZED NYQUIST CRITERION 

A waveform v(t) is said to satisfy the Nyquist criterion 1 for the 
signal interval T, if 

v(nT) = 5„o n = 0, ±1, ±2, ••• . (10) 

Denoting the Fourier transform of v(t) by V(f) (upper case letters 
will be used throughout to denote the Fourier transforms of the func- 
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tions represented by lower case letters), we can state that (10) is true, 
if and only if, 

KM'-?)- 1 - (u> 

This is easily shown using Poisson's sum formula (Papoulis) 3 

ft, *(' - t) - .t«""* , "*W- (12) 

If we associate <£(£) with v(t) then (10) implies and is implied by 

(11). 

Our constraint that the v mk (t) satisfy (8) requires not only that the 
v mm {t) satisfy (10) but also that the M(M - 1) waveforms v mK (t) (m s* k) 
be zero at t = nT. We refer to (8) as the generalized Nyquist criterion. 
The equation analogous to (11) is 

££.*-(' -?)-■- (13) 

This will be used interchangeably with (8) in solving the optimiza- 
tion problem. Since the {V mk (f)} can be checked almost by inspection 
to see if they satisfy (13), the equation is very simple to use. 

III. THE CONSTRAINED OPTIMUM RECEIVER 

The object of this section is to determine the linear receiver which, 
subject to the constraints of (8), minimizes the error expression (9). 
Because of the constraint of (8) we have 

b nk - a„, = f v(x)M' ~ *) dx (14) 

so that the error becomes 

•/ = Tf £ [ " W m (f)W*(f)N(f) df (15) 

which is independent of n. 

We are now left with the interesting variational problem of mini- 
mizing J with respect to all linear receivers W k (/) such that 



f^£ «-(f-?W -?) = ««- 



(16) 



i.e., which satisfy the generalized Nyquist constraint. In order to do 
this, we vary each W k (f) by an amount c r fc (/), where the r fc (/) must 
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be such that (16) is still valid. We require 

= \ m t *.(/ - §)w(f - 1 ) + . m ± R n (f - §)r.(/ - 1 ) = 5mA . 

fc, to - 1,2, ••• ,M (17) 
so that r ; ,-(/) must satisfy the condition 

j£ *.(/ - |)r*(/ - f) - m, k = 1, 2, • • • , M . (18) 
The error with variations becomes 

= Yi % /_" H'*(fl "■'**(*)#(/) # 

+ fi%F FMW*® + rtO)W»<0TO # 

+ ^ Z £ r*(/)n(/Wfl #. (19) 

,7(0) is minimum if (the u'i t (t) arc constrained to be real) 

f r k (f)W%(f)N(f) (If = (for each Ic = 1 ■■■ M), (20) 

where Tfc(/) must satisfy (18). 

In order to solve for W k (j) we manipulate (20) as follows: 

•'-00 

- Z r ' r,(/)irt(/)A-(/)r// 

<i--oo •'-l/2T+a/T 
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Comparing (21) with (18) it can be recognized that (21) is satisfied 
by aW k (f) such that 

wm- tj$*Jfi *-i. ... .Jr. (22) 

where the Z ak (f) are arbitrary periodic functions of / with period 1/T. 
In order to completely specify W k (f) we must determine the Z ek (f). 
Substituting (22) into (16), we obtain 

- «♦.(/ - c 






. . . B.(/-fH/-f) 

= \ Z BJIi S v i \ - «- < 23 > 

I ... .--. ^ _ ^ 

m, ft = 1,2, • • • , 7rc 
since Z c7 ,(/) is periodic with period 1/T. Let 

L.,(/) = ^£4-fH/-f)A(/-?) (24) 

then (23) becomes 

I: Z LJtoZJfi = Ku m,lc= 1,2 •■■ m (25) 

or in matrix form 

£(/)£(/) - /, ( 2r ») 

where 

W = [L t M 

and 

Z(j) - [Z„(/)] 

are M by Af matrices. 

Thus, we have, if L is nonsingular for all /, that 

Z(j) = [L(f>r (27) 
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SO that | L | 9* is :i necessary and sufficient condition for a solution to 
exist. W k (J) is now completely specified and a realization of the optimum 
constrained receiver is show: i in Fig. 2. 

A simple expression for the resulting mean square error is obtained 
from a manipulation similar to that of (21) : 



Jo., =J?ZI Z mm (f) df. 

-M m _i J-1/2T 



(28) 



IV. EXAMPLES 



In this section examples are presented which demonstrate the ease 
with which the generalized Nyquist criterion can be used to design 
waveforms without intersymbol interference or crosstalk. 

4.1 Example 

We start out by making the simplifying assumption that N(f) = 1. 
In addition, if the transmitted waveforms {£„,(/) }f are chosen such 
that R m (j), where R m (J) = S m (f)H(f) satisfy the equation 



q Z R m [1 ~ %)Rl[1 -§) = d m 8 mk , (29) 

then a solution exists since the L matrix becomes a diagonal matrix 

"rf, 



L = Id; 



(1 = 



(30) 



R,*(f] 
N(f) 



z„(f) 



ZiMlf) 



N(f) 



Zui(fJ 



z MM (f) 




Fig. 2 — A realization of the optimum constrained receiver. 
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and 



Z = L~ l = 



1/* 




i/ch 



(31) 



with the resulting error 






(32) 



Under these conditions the outputs of the matched filters satisfy the 
generalized Nyquist constraint except for scale factors and the Z cm (/) 
functions need only perform the appropriate scaling. We consider next 
two cases where (29) is satisfied. 

4.2 Case I 

Only the case of M = 2 is presented here in detail although other 
values of M can similarly be handled. 

First, note that since matched filters are used the actual phases of 
the Ritf) are not important since the output depends only on phase 
difference between Rdf) and Rj(f). We use the phase of BAf) as a 
reference phase. 



Rx(f) = 



ff»(fl = 



ice 



,/#i(/) 



I / I £ V?' 

,o, | / 1 > i/r 



\cc 



In, 



/ I ^ 1/T 
f I > l/T 



where A0(/) = A0( — /) ±tt for | / | ^ 1/7'. The sign is chosen so that 

|A0(/) I ^ r. 

To simplify matters we can choose 



A<K/) - 



tt/2, 
-t/2, 



/>0 
/<0 



so 



R^m%D = 



fo-iuin 



10, 



/ 1 > i/r 
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jc', Og/S 1/T 
jc~, -\,T ^ j < 

0, elsewhere. 



Therefore, 

jt m *«(/ - %)r**(i - f) = P* - Jc 2 = (Fig. 3). 
Similarly, 



and 



so 



and 



zM/-fM/-f -o 
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2c 2 n 
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2c 
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Fig. 3 — Case I transforms at the output of the matched filter. 
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In the time domain we have 



vM =c 2 (cos^- l)/x/ 



and it is easily seen that 

v l2 (nT) = for n - 0, ±1, ±2, • • • . 

4.3 Case II 

Consider a set of band-limited frequency multiplexed signals \R m (j) ) f . 
The bandwidths are (1 + y m + y m -i)/T where the 7m (0 ^ 7m ^ 1) 
are parameters associated with the excess rolloff bandwidth, and the 
signals are separated in frequency by 1/T hertz so that the waveforms 
overlap the adjacent signals only. As in Case I, the actual phases are 
unimportant because of the matched niters so only phase differences 
m (/) from a reference phase <£(/) will be important. 

R m (J) = | RJf) | exp Mi) + 4> m (f)] 

R~(f)RZ +i (f) = \R~(f)R-*i(J) I «P jfe.0) - *- + i(01. 

We define roll-off characteristics as a real function Q m (f) such that 



Q m (f) = for | / | > jfi 



and 



Q m (J) = -Q,(-1); for 
We can specify the R m (/) as follows: 



J l = 27' 



= ,,„Vrect (lil^5) + 0.(|/| + i-f) + Q m+ {- | / 

A,„(/) = *.(/) - *»-i(/) 
and 

A,„(/) = A„,(-/) ± 7T. 
With the R,,, (/) specified it is easily checked that 

= cL 



1 m 
2f~~f 



*J/-f 
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R„.(f)R„,->(1) = I /U/) I I #.->(/) | exp [- ;A„,(/)] 



< • I- !/ - VL 1 r^)^ './X'/il 



where B m (f) is an even real function with bandwidth 2y m _i/T. 

#,(/) - V(?»(i / i)[i - Q„(-i/i)r 

Wo can specify A m (/) as 



A„,(/) = < 



7T 



, />0 



7T 



without really restricting ourselves. The resulting R m R%_ l is shown 
in Fig. 4. Looking at Fig. 4, we see by inspection that the [R m ] " satisfy 
(29). 
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Fig. 4 — -Case II transforms at the output of the matched filter. 
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APPENDIX 

The Optimum Mean Square Error Receiver 

In this appendix the optimum mean square error receiver is obtained 
and it is shown that as S/N -> oo this receiver and the optimum con- 
strained receiver of Section III converge to the same receiver when the 
{a nk } are stationary. 

The general expression for the mean square error is 

J n = h £ E U- " 2 E Z a nm a pk v km (nT - pT) 



(33) 



+ EIZE a pk a Ti v km {nT - pT)v im (nT - rT) 

- 2o nm I ri(x)w m (nT - x) dx 

+ 2 £ £) a pk v km (nT - pT) f° v(.v)w m (nT - x) dx 

p=-oo 6 = 1 J -to 

+ [[ v{x)t](y)w m (nT - x)w m (nT - y) dx dy 
= h £ \pZ. - 2 £ £ pZv km (nT - V T) 

Wl »/ = ! _ p=-oo fc=l 

IZZEI P i;v km (nT - P T)v im (nT - rT) 

p = -oo k = \ r = -oo 1 = 1 

+ f NWWJflWt® dtj , 

where 

P Z = E{a nm a pk \. (34) 

Since the {a nk } are stationary we can write 

»,J> _. ("• 

Pmk = Pmk 



(n-p, (35) 
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Defining 

M mk (J) = Z P ':l exp(-j2tr1nT), (36) 

n = - x> 

then J n can be written as 
J = 77 Z U2 " 2 Z P -!/„.,(/) r?,„(/) d/ 

•"' m-l L *-l •'-* 

+ Z Z f M ti (j)VL(f) \ Z vJf - f) d/ 

A = l 1 = 1 •'-oo -* a=-ao \ l I 

+ /" W m <j)WZW(fl rf/] (37) 

which is independent of n so the index has been dropped. 

Using variational calculus we obtain as a necessary condition on the 
optimum { W m (l) \ i" that they satisfy the equations 

Z M(f>[t M mk (j) \ Z Ri(f ~ |)TF,(/ - f,) - M mk (f)~\ 

+ W k (W(1) = fc = 1,2, ■•■ ,M. (38) 
The solutions for the \W k (J)\" are 

W k (f) = Z ^ Y ek (i) k = 1, 2, • ■ - , M, (39) 

where the Y ck (j) are periodic functions of / with period \/T. In order 
to see that the \W k {j)\f of (39) satisfy (38) for the appropriate deter- 
mination of the {Y ck (j)\, substitute (39) for W k {j) in (38) to obtain 

m r m l °° 

Z RS0) Z M mi (f) r Z AM/ - (a/T)] 

,,, I L 1-1 -* O--S0 

f tf«f/- (a/T)] m , "1 

+ Z R*A1) Y ek (J) = fc = 1, 2, • • • , M (40) 

e = l 

or since the Y cl: (/) are periodic 



Z to/) h ',,,,(/) - .1/ (/) + Eif ..,(/) E4(i)L(D 

ml L 1=1 c = l 



= 



/„• = 1, 2, ••• , M, (41) 
where L (c (/) is as defined in (24). 
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Defining M(f) and Yij) as matrices whose elements are, respectively, 
Mij(f) and Y tJ (f) and a column vector R(f) whose elements are R m (f) 
(41) can be written as 

R(f) T (Y(j) - M(j) + M(f)WY(f)) = 0, (42) 

where L(f) is as previously defined. Unless R(f) — we require that 

(/ + ML) Y = M (43) 

Y = (/ + MVf x M. (44) 

With Y so specified (39) satisfies (38) and the resulting mean square 
error is 

Manipulating as in (21) and using the periodicity of M(f) and Y(f) 
we then obtain 

m M r \/2T r M M 1 

J « - T7 E M mm {\) -EZ M mk (f)L* c {j)YUf) if. (46) 

IVl m = \ J-1/2T L k-1 e-1 -I 

Lastly, recognizing that the integrand is Y* m (J) we get 

rp M pl/2T 

J opt = y f Z Y* m (j) df. (47) 

Finally, we wish to show that the optimum and constrained optimum 
receivers approach the same limit as S/N -» <x> . 

We define U to be the resulting L matrix when the S/N is unity, 
and we write for any other S/N 

L = aU, (48) 

where a is proportional to the signal energy. Since both receivers are 
of the same form, we need only show that Y -* Z as a — > oo . 

Y = (ML + I)~ l M 

= (aMU + l)~ l M 

= [(l/^CT'AT 1 - (l/a 2 )(U- l M-y + (l/a 3 )(CT , M- , ) a - ..-]Jlf 

= (l/a)U~ l +0(1/ 'a 2 ), (49) 

where 0(l/a 2 ) indicates terms dropping off at least as fast as 1/a. 2 As 
a -> oo the terms of order 1/a 2 become negligible with respect to the 
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1/a term. Using the fact that L~ x = \/a If- 1 , we obtain the result 

lim Y = XT* = Z, (50) 

a-«DO 

and the two receivers converge and the constrained optimum is opti- 
mum. 
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